An Information Theoretic Model for Database Alignment

As with many large organizations, the Government's data is split in many different ways and is collected at different times by different people. The resulting massive data heterogeneity means that government staff cannot effectively locate, share, or compare data across sources, let alone achieve computational data interoperability. The premise of our research is that it is possible to significantly reduce the amount of manual labor required in database wrapping and integration by automatically learning mappings in the data. In this research, we applied statistical algorithms to discover column correspondences across environmental databases. We have seen particular success in an information theoretic model, which we call SIfT, which performs data-driven column alignments. We have applied SIfT to mapping Santa Barbara and Ventura County Air Pollution Control Districts’ 2001 and 2002 emissions inventory databases with the California Air Resources Board statewide inventory database. The application of SIfT yielded 75% precision and 72.2% recall on the column alignment task. On a task of integrating new district data with the statewide database, we achieved 55% accuracy for Ventura County and 59% accuracy for Santa Barbara County.